Skip to content
This repository has been archived by the owner on Nov 23, 2023. It is now read-only.

feat: update dataset catalog with version #741

Merged
merged 18 commits into from
May 31, 2021
Merged

feat: update dataset catalog with version #741

merged 18 commits into from
May 31, 2021

Conversation

MitchellPaff
Copy link
Contributor

No description provided.

@MitchellPaff MitchellPaff changed the title Feat/update dataset catalog with version feat: update dataset catalog with version May 25, 2021
@MitchellPaff MitchellPaff force-pushed the feat/update-dataset-catalog-with-version branch 8 times, most recently from 082773b to e7f6e85 Compare May 31, 2021 02:34
@MitchellPaff MitchellPaff marked this pull request as ready for review May 31, 2021 04:46
@MitchellPaff MitchellPaff requested a review from l0b0 May 31, 2021 04:46
l0b0
l0b0 suggested changes May 31, 2021
View reviewed changes
backend/populate_catalog/task.py Show resolved Hide resolved
backend/populate_catalog/task.py Outdated Show resolved Hide resolved
backend/populate_catalog/task.py Outdated Show resolved Hide resolved
backend/populate_catalog/task.py Outdated Show resolved Hide resolved
backend/sqs_message_attributes.py Show resolved Hide resolved
tests/test_populate_catalog.py Outdated Show resolved Hide resolved
Mitchell Paff and others added 17 commits June 1, 2021 10:27
refactor: Pass dataset prefix earlier in pipeline to avoid
4c7989b 
Refactored the input to pipeline to include the dataset prefix.
This is to avoid fetching this from the database multiple times during
the pipeline.
fix: Remove unnecessary input into pipeline steps
5585db8
feat: add new dataset version to dataset catalog
74073c5
fix: added tests
03accfa
refactor: parametrize tests
750221f
refactor: implement custom layout strategy
66cee56
refactor: add test for exception case
3003b06
refactor: fix rebase
f8cf40f
refactor: add stronger assertin to update_catalog test
493267a
fix: fix lock file
abdf13f
fix: fixed broken import from other lambda
85c5a8c
refactor: Strengthen assertions on sqs tests
faf0084
fix: Fix inconsistency sqs and lambda casing of message attributes
7ab06dc
fix: Fix table access exception and refactor custom layout strategy
85e7d7a 
Refactored the input to pipeline to include the dataset prefix.
This is to avoid fetching this from the database multiple times during
the pipeline.

fix: infra
refactor: Expand usage of STAC custom layout strategy
84a4ca7 
Added extra test for when adding a dataset version with Item metadata
and expanded the usage of our custom layout strategy to avoid overwriting
existing links when creating a new dataset.
fix: Add a self ref to root catalog
e686369
refactor: Address PR comments
c557a22
@MitchellPaff MitchellPaff force-pushed the feat/update-dataset-catalog-with-version branch from 1e157b7 to c557a22 Compare May 31, 2021 22:27
@MitchellPaff MitchellPaff requested a review from l0b0 May 31, 2021 22:51
@kodiakhq kodiakhq bot merged commit 3e7ba19 into master May 31, 2021
@kodiakhq kodiakhq bot deleted the feat/update-dataset-catalog-with-version branch May 31, 2021 23:23
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@l0b0 l0b0 l0b0 approved these changes

Assignees
No one assigned
Labels
automerge
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@MitchellPaff @l0b0